workaround for signal during grpc test framework teardown - #1691

No actionable comments were generated in the recent review. 🎉

_{Comment @coderabbitai help to get the list of available commands.}

CI Test Summary

✅ All 31 test job(s) passed.

🔔 Hi @anandhkb, this pull request has had no activity for 7 days. Please update or let us know if it can be closed. Thank you!

If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.

I traced this through the nightly path. Short version: it won't break nightly, but the shell-side heuristic opens a false-green hole that's wider than the bug it works around.

Checked and fine

• --gtest_list_tests piping. This was my main worry: run_ctests.sh:130 pipes the binary's stdout into junit_helpers.py gtest-list in the nightly crash path, and piped stdout is block-buffered — std::_Exit skipping the stdio flush would have emptied that listing and turned every nightly crash-retry into FAILED: Could not list tests. It doesn't, because ListTestsMatchingFilter ends with an explicit fflush(stdout) (googletest gtest.cc:6426).
• Log tail and XML survive _Exit. PrettyUnitTestResultPrinter::OnTestIterationEnd ends in fflush(stdout) (gtest.cc:3843) and XmlUnitTestResultPrinter::OnTestIterationEnd does fclose(xmlout) (gtest.cc:4093). Nothing is left buffered.
• Coverage. No gcov on C++ (test.yaml:62 sets run_codecov: false; coverage is Python-only), so the usual "_Exit drops coverage counters" objection doesn't apply.
• Orphaned servers / port leaks. Your safety claim holds — every suite's TearDownTestSuite does s_server_.reset(), which runs ~ServerProcess → stop() → kill(-pgid_, ...), all inside RUN_ALL_TESTS.
• compute-sanitizer. test_cpp_memcheck.sh:60 only runs ROUTING_TEST/ROUTING_GES_TEST, so this binary never goes through it.

The concern

accept_post_pass_teardown_crash lives in run_gtest_with_retry, which loops over all "${GTEST_DIR}"/*_TEST (run_ctests.sh:215). But this binary now _Exits and can never reach that path — so the heuristic exists purely to swallow signal-deaths in other binaries, which is exactly where we have no diagnosed root cause.

The masked window is narrower than I first assumed, to be fair: global Environment::TearDown() runs at gtest.cc:6119, before the XML write at 6141, so a crash in a gtest global environment still produces no XML and is still caught. What's masked is strictly "after gtest wrote XML" — tail of UnitTest::Run, return from main, and static/atexit destructors. That's still precisely where a genuine libcuopt/RMM/CUDA static-destruction-order bug would surface, and it's user-facing: anything linking libcuopt would abort at exit the same way.

It also leaves no telemetry. nightly_report.py:130 collects results via rglob("*.xml") only — it never reads the run log, so the WARNING: line reaches neither the report, the history file, nor the dashboard. Between that and _Exit, the grpc double-free now produces zero signal, which makes "investigate separately" start with no frequency data on whether it's getting better or worse.

Suggestion

Keep std::_Exit(rc) — it's targeted and verifiably safe for this binary. For the shell heuristic, either:

1. Scope it to this binary (name check or an env var set only for GRPC_INTEGRATION_TEST), so an exit-time abort anywhere else still fails nightly; or
2. If it stays global, have it write a *-teardown-crash.xml marker via the existing write_crash_xml in crash_helpers.sh as a passing-but-recorded entry, so nightly_report.py can track the occurrence rate instead of the signal vanishing.

Minor

• The new block under [ -z "${tests_to_retry}" ] is unreachable-when-true: getting there requires a signal death, and accept_post_pass_teardown_crash already evaluated the identical xml_reports_all_passed condition and returned 0. Harmless, but it reads like a second safety net that can never fire.
• std::_Exit is declared in <cstdlib>, which grpc_integration_test.cpp doesn't include (it has <cerrno>, <cmath>, <filesystem>, <fstream>, <mutex>, <sstream>). It compiles today only transitively — worth adding explicitly.
• xml_all_passed counts a <skipped/> testcase as a pass. Probably fine here, just noting it.

🔔 Hi @anandhkb, this pull request has had no activity for 7 days. Please update or let us know if it can be closed. Thank you!

If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.

/merge